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In this paper, we provide new theoretical results on the generalization properties of learning algorithms 

■ for multiclass classification problems. The originality of our work is that we propose to use the confusion 
matrix of a classifier as a measure of its quality; our contribution is in the line of work which attempts 

■ to set up and study the statistical properties of new evaluation measures such as, e.g. ROC curves. In 
^ , the confusion-based learning framework we propose, we claim that a targetted objective is to minimize 

the size of the confusion matrix C, measured through its operator norm \\C\\. We derive generalization 



OO , bounds on the (size of the) confusion matrix in an extended framework of uniform stability, adapted to 

■ the case of matrix valued loss. Pivotal to our study is a very recent matrix concentration inequality that 
generalizes McDiarmid's inequality. As an illustration of the relevance of our theoretical results, we show 

■ how two SVM learning procedures can be proved to be confusion-friendly. To the best of our knowledge, 
^ 1 , the present paper is the first that focuses on the confusion matrix from a theoretical point of view. 

^ ■ Keywords: Machine Learning, Stability Generalization Bounds, Confusion Matrix, Non-Commutative 

Q i Concentration Inequality, Multi- Class 

1 Introduction 

I Multiclass classification is an important problem of machine learning. The issue of having at hand sta- 

■ tistically relevant procedures to learn reliable predictors is of particular interest nowadays, given the 
I widespread need of such predictors in information retrieval, web mining, bioinformatics or neuroscience. 

■ Yet, the literature on multiclass learning is not as voluminous than that of binary classification, whereas 
^3 ! this problem raises questions from the algorithmic, theoretical and practical points of view. One of the 
04 ■ prominent questions is that of the measure to use in order to assess the quality of a multiclass predictor. 

Here, we develop our results with the idea that the confusion matrix is a performance measure that 
deserves to be studied as it provides a finer information on the properties of a classifier than the mere 
_ misclassifica tion ra t e. M ore precisely, building on very recent matrix-based concentration inequalities 

■ provided bv iTropo ( 2011 ) — sometimes referred to as noncommutative concentration inequalities — we 
I establish a stability based framework for confusion- aware learning algorithm. In particular, we prove a 

generalization bound for confusion stable learning algorithms and show t hat there exist such algorithm s 



in the literature. In a sense, our framework and our results extend those of lBousauet and Elisseeffl ()2002l ). 
which are designed for scalar loss functions. To the best of our knowledge, this is the first work that 
establishes generalization bounds for confusion matrices. 

The paper is organized as follows. Section [2] describes the setting we are interested in and motivates 
the use of the confusion matrix as a performance measure. Section [3] introduces the new notion of stability 
that will prove essential to our study; the main theorem of this paper, together with its proof, are provided. 
Section m is devoted to the analysis of two SFMprocdures in the light of our new framework. A discussion 
on the merits and possible extensions of our approach concludes the paper (Section [S]). 
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2 Confusion Matrix Awareness 

2.1 Notation 

As said earlier, we focus on the problem of multiclass classification The input space is denoted by X and 
the target space is 

3^ = {i,.-.,Q}. 

The training sequence 

z = {z, = (x„r,)}^i 

is made of m identically and independently random pairs Zi ~ {Xi,Yi) distributed according to some 
unknown (but fixed) distribution D over X x y. The sequence of input data will be referred to as 
X = {Xi}'^i and the sequence of corresponding labels Y = we may write Z = {X,Y}. The 

realization of Zi = {Xi,Yi) is Zi = {xi,yi) and z, x and y refer to the realizations of the corresponding 
sequences of random variables. For a sequence y = {yi, ■ ■ ■ ,ym} of m labels, mq{y), or simply when 
clear from context, denotes the number of labels from y that are equal to g; s(y) it the binary sequence 
{si(y), . . . , SQ{y)} of size Q such that Sq[y) = 1 if g e y a-nd Sq{y) = otherwise. 

We will use Dx\y for the conditional distribution on X given that Y = y] therefore, for a given sequence 
y = {yi, . . . , ym} G 3^"*, Dx\y = ®^iDx\yi is the distribution of the random sample X = {Xx, . . . , X^} 
over X^m such that Xj is distributed according to Dx\yi ; for g S 3^, and X distributed according to Dx\y, 
Xq ~ {Xi-^ , ■ • ■ , ^imq } denotes the random sequence of variables such that Xi^ is distributed according 
to Dx\q- IE[-] and Ejf|j,[-] denote the expectations with respect to D and Dx\y, respectively. 

For a training sequence Z, Z* denotes the sequence {Zi, . . . Zi^i, Z\, ^i+i, . ■ • , ■Z^m} where Z\ is dis- 
tributed as Zi\ Z^' is the sequence {Zi, . . . Zi^i, Zi^i, . . . , Zm} — these definitions carry directly over 
when conditioned on a sequence of labels y (with, henceforth, y'^ = Vi)- 

We will consider a family % of predictors such that % <Z {h : h{x) £ M9 , Vx e X}. For h eH, hq eR'^ 
denotes its qth coordinate. Also, £ = {iq)i<q<Q is a set of loss functions such that: £q : H x X x y ^ M+. 

2.2 Confusion Matrix 

We here provide a discussion as to why minding the confusion matrix or confusion loss (terms that we will 
use interchangeably) is crucial in multiclass classification. We also introduce the reason why we may see 
the confusion matrix as an operator and, therefore, motivate the recourse to the operator norm to measure 
the 'size' of the confusion matrix. As the definition of the confusion loss in the online learning framework 
is less usual and a bit more intricate than its definition in the batch scenario, we develop the discussion 
here within the batch setting — obviously, the argument carries over to the online learning framework. 

In many situations, e.g. class-imbalanced datasets, it is important not to measure the quality of 
a predictor H on its classification error PxY{h{X) ^ Y) only; this may lead to erroneous conclusions 
regarding the quality of h. Indeed, if, for instance, some class q is predominantly present in the data 
at hand, say P{Y = q) ^ 1 — e, for some small e > 0, then the predictor /i,„aj that always outputs 
^maj(a^) ~ q regardless of x has a classification error lower thant e. Yet, it might be important not 
to classify an instance of some class p in class q: in the context of classifying mushrooms according 
to the categories {hallucinogen, poisonous, innocuous}, the consequence of predicting innocuous (the 
majority class) instead of hallucinogen or poisonous might be disastrous. 

As a consequence, we claim that a more relevant object to consider is the confusion matrix which, 
given a binary sequence s = {si ■ ■ ■ sq} € {0, is defined as 

C,{h):= ^X\gm,X,q), 

q:s, = l 

where, given an hypothesis h £ H, x G X , y E y, L{h, x, y) = {Uj)i<i,j<Q G MP^'^ is the loss matrix such 
that: 

I. \ ^3ih,x,y) \ii = yimdi^j 
' [0 otherwise. 

Note that this matrix has at most one nonzero row, namely its zth row. 
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For a sequence y G of m labels and a random sequence X distributed according to Dx\y, the 
conditional empirical confusion matrix Cy{h,X) is given by 



where 



m ^ 

Lg{h,X,y) := — V 



For a random sequence Z = {X, Y} distributed according to D™, the (unconditional) empirical confusion 
matrix is given by 

^X\YCY{h,X) =Cs{Y){h), 

which is a random variable, as it depends on the random sequence Y. For exposition purposes it will 
often be more convenient to consider a fixed sequence y of labels and state results on Cy{h,X), noting 
that 

^X\yCy{h,X) ^Cs(y){h). 

The slight differences between our definitions of (conditional) confusion matrices and the usual defini- 
tion of a confusion matrix is that the diagonal elements are all zero and that they can accomodate any 
family of loss functions (and not just the — 1 loss). 

A natural objective that may be pursued in multiclass classification is to learn a classifier h with 'small' 
confusion matrix, where 'small' might be defined with respect to (some) matrix norm of Cs{h). The norm 
that wc retain is the operator norm that we denote j| • || from now on: for a matrix M, \\M\\ is defined as 

\\M\\ ^max " " , 

■c^O \\v\\2 

where ||-||2 is the Euclidean norm; is merely the largest singular value of M — note that = 

A nice reason for focusing on the operator norm is that Cs{h) is often precisely used as an operator 
that acts on the vector of prior distributions 

TV = [p(y = i)---p(r = Q)]. 

Indeed, a quantity of interest is for instance the risk Re{h) of h, with 



R,ih) := !|7rTCi(/i)||i = J2 ^x\pe<i(.h,X,p)Trp 



p,g=l 

= Ey |^Ex|y^,(/l,X,r)| 

It is interesting to observe that, V/i,V7r: 

0<i?^(/i) = ||7rCi(/i)||i=7r^Ci(/i)l 

< v^||7r^Ci(/^)||2 = ^\\Cj{h)'K\\^ 

< VQ\\Cl(h)\\ M2 

< VQ\\Cj{h)\\ = v/Qj|Ci(/^)||, 

where we have used Cauchy-Schwarz inequalty in the second line, the definition of the operator norm on 
the third line and the fact that ||7r||2 < 1 for any tt in {A G M*^ : Ag > 0, = 1}. 

In addition to support the use of the operator norm, this also says that bounding the norm of the 
confusion loss is a good way to control the risk of h as well (independently of the prior distribution, see 
discussion below). 
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3 Deriving Stability Bounds on the Confusion Matrix 

One of the most prominent issues in learning theory is to estimate the real performance of a learning 
system. The usual approach consists in studying how empirical measures converge to their expectation. 
In the traditional settings, it often boils down to providing bounds describing how the empirical risk 
relates to the expected one. In this work, we show that one can use similar techniques to provide bounds 
on the operator norm of the confusion matrix. 



3.1 Stability 

Following the early work of VapnikI ( 1982I ). the risk has traditionally been estimated through its empirical 



measure and a measure of the complexity of the hypothesis class such as Vapnik-Chervonenkis (VC-dim), 
fat-shattering dimension or Rademacher complexity. During the last decade, a new and successful approach 
based on algorithmic stability to provide some new bounds has emerged. One of the highlights of this 
approach is the focus on some properties of the learning algorithm at hand, instead of the characterization 
of the hypothesis class. Roughly, this makes it possible to take advantage from the knowledge on how a 
given algorithm actually explores the hypothesis space , often leading to tighter bounds. 

The main results in iBousauet and Elisseefj (|2002h were obtained using the following definition of 
uniform stability. 

Definition 1 (Uniform stabilitv iBousauet and Elisseefi ( 2002 )). An algorithm A has uniform stability /3 
with respect to the loss function I if the following holds: 



yS e Z"\yi £{!,..., m}, 11/(^5, .) - 1{A, 



■ 



< 



Following the same approach, we now focus on the generalization of such results for confusion matrices. 
We introduce a new definition of confusion stability. 

Definition 2 (Confusion stability). An algorithm A is confusion stable with respect to the loss matrix L 
ifVq G y, there exist a /3q decreasing as — such that A has fiq uniform stability with respect to the loss 

L 



3.2 Non-Commutative McDiarmid 

In IBousauet and Elisseefi (l2002l) . the authors make use of some well-known concentration in equalities to 
derive bounds. More specifically, they use a variation of Azuma's inequality, due to McDiarmid lMcDiarmid 
- 1 describes how a scalar function of independent random variables (the elements of our training 
set) normally concentrates around its mean, with variance depending on how changing one of the random 
variables impacts the value of the f unctio n. 

Some more recent work iTropd (j201ll ) extends McDiarmid's inequality to the matrix setting. For the 
sake of self-containedness, we recall this non-commutative bound. 

Theorem 1 (Matrix bounded difference ( Tropnl ( 2011 ). corollary 7.5)). Let S and S'' be defined as above, 
and let H be a function that maps m variables to a self-adjoint matrix of dimension Q. Consider a 
sequence {Ai} of fixed self-adjoint matrices that satisfy 



{H{S) - H{S'^)y 4 Al, 



(1) 



where Zi and Zi range over all possible values of the space Zi it belongs to, for each index i. Then, for all 
t > 0, 

n^,na4H{z) - EH{z)) >t}< ge-*'/^'^', 
where z — {Zi, . . . , Zm) and a"^ := || Af\\ . 

One may notice that H has to be applied on a function mapping to self-adjoint matrices. Unfortunately, 
our confusion matrices are real- valued but are not symmetric. However, we can make use of a dilation 
technique to overcome this. 

Namely, instead of working directly on a non-self-adjoint matrix A, we will build a self-adjoint dilated 
matrix D{A) : 



S)(A) 





A* 
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where A* denotes the adjoint of A. 

The choice of such a dilation is directly motivated by the following property, recalled in iTropd (|201l[) . 

Lemma 1. If X is the largest eigenvalue of'D{A), then A is the largest singular value of A. 

As a direct consequence, any result on the largest eigenvalue of a dilated matrix ©(A) directly applies 
to the operator norm ||A|| of ^. 



3.3 Stability Bound 

Theorem 2 (Confusion bound). Let A be a learning algorithm. Assume that all the loss functions under 
consideration take values in the range [0;M]. Let y g 3^™ be a fixed sequence of labels. 

If A has confusion stability (3 with respect to all loss matrices Lq, for q € y, then, Vm > 1, V(5 € (0, 1), 
the following holds, with probability 1 — S over the random draw of X ^ L)x\yj 



Cy{A,X)-Cs(y){A) <2^/3g + Qj81n(^) 4^^^/?, 



M 



Q 



As a consequence, with probability 1 — 5 over the random draw of Z ^ D" 



CY{A,X)-CsiY){A) <2^/3, + QW81n(^) [iV^^l3min + M 



Sketch of proof. The complete proof can be found in the following subsection. We proceed in three steps 
to proof the first bound. To start with, we know that by the triangle inequality 



\\CiA,X)-C,^y)iA)\\ 



J2{L,{Az:Z)-ExLq{Az,Z)) 



< J2 \\Lq{Az, Z) - ExLqiAz, Z) 



(2) 



Using standard uniform stability techniques, we bound each summand with probability 1 — S/Q. 

Then, using the union bound we have a bound on ||C(,4, — Cs(j^)(^)|| that holds with probability 
at least 1 — (5. 

Finally, recoursing to a simple argument, we express the obtained bound solely with respect to mmin- 
In order to get the bound with the uncondional confusion matrix Cs(y){A) it suffices to observe that 

for any event £{X, Y) that depends on X and Y, such that for all sequences y, ¥x\y{£{X , y)} < S, the 

following holds: 

¥xy{£{X, Y)) = ExY {l{£iX,Y}} } 

= Ey {lEx|yI{£(x,y)}} 

= J2^x\Yk£ixx)}^YiY = y) 

V 

= J2Px\y{S{X,y)}PY{Y = y) 
y 

<Y,5¥y{Y = y)^5, 



which gives the desired result. 



□ 



Remark 1. It is straightforward to directly obtain a bound on ||Cs(y)(^)|l and |lCs(y)(w4)jl by using the 
triangle inequality — |1-B|1| < ||j4 — _B|| on the bounds given in the theorem. 



Technical Report V 1.0 



5 



P.Machart, L. Ralaivola 



Confusion Matrix Stability Bounds for Multiclass Classification 



3.4 Proof of Theorem [2] 

Proof. To ease the readability, we introduce some additional notation: Z, Z', Z^' will refer respectively to 
{X,y},{X\y^,{X\\y\^li,nd 

C, = Exi,Li^z,X, q), 4 = LgiAz, X, y), 
CI = ExiM^Z' ,X,q), L^{Az^ , X\y'), 

=Ex|g£(^z\M^,9), =L,{A:,^,,X\\y\'). 

After using the triangle inequality in ([2]), we need to provide a bound on each summand. To get the result, 
we will, for each q, fix the Xk such that yk ^ q and work with functions of variables. Then, we will 
apply Theorem [1] for each 

To do so, we will bound the differences 

\\H,{X„y^)-H,{X\,y\)\\. 

Note that 

\\H,{X„ y^) - H,{X\,y\)\\ = - - T,{C\) + 

= \\Cq - Cq~ > 

Step 1: bounding \\Cq — C^q\\- We can trivially write: 



Cq-Cq~Cq+ Cq\\ < \\Cq - £^11 + \\£q - Cq\ 



\Cq-Cl\\<\\Cq-CY\\+\\Cq-CY 



Using the /3g-stability of A: 



\\Cq-C\q'\\ = \\Exiq[L{Az,X,q)-L{Azv,X,q)] 
< Ex\q \\L{Az,X, q) - LiAzv , X, q)\\ 

and the same holds for WCq-CYW, i.e. - cY\\ < Thus, we have: 

\\Cq-Cl\\<2Pq. 



(3) 



Step 2: bounding \\Cq — This is a little trickier than the first step. 

\\Cq-Cl\\ = \\Lq{Az,Z)-Lq{Az^,Z'')\\ 

^ ' J2 {L{Az,Xk,q)^LiAz^,Xk,q))+L{Az,X,,q)-LiAz^,Xl,q) 

k:k^i,yk=q 

1 



< 



' k:k^i,yk=q 



J2 [L{Az.,Xk,q)-L{Az^,Xk,q)) +— L{Az,X,,q)-L{Az^,X[,q) 



Using the /3g-stability argument as before, we have: 



^ [L{Az,Xk,q)~L{Az^,Xt,,q) 
k:k^i,Vk=q 



< \\H-Az,Xk,q)~L{Az^,Xk,q)\\ 

k:k^i,yk=q 

k:k:^i,yk=q 



On the other hand, we observe that 

L{Az,X,,q)^LiAz^,Xl,q) <yOAf. 
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Indeed, the matrix A := L{Az,Xi, q) — L{Azi , X'^, q) is a matrix that is zero except for (possibly) its qth 
row, that we may call Sq. Thus: 

||A1| = sup ||At;||2= sup \\Sq ■ V\\ = \\6q\\2, 

where v is a vector of dimension Q. Since each of the Q elements of 6q is in the range [—M; M], we get 
that |l<5,|i2 < VQM. 

This allows us to conclude that 

!l4-£^||<2/3, + ^ (4) 
^ mq 

Step 3: Applying Matrix McDiarmid Combining ([3]) and that we just proved, we have that, 
for all i such that Ui = q 

[Hq{Zq) - Hq{Z\)f ^ U^q + ^) /. 

Therefore, Theorem [T] may be applied on Hq{Xq, yq) ~ 'Zi{Cq — Cq) with 

2 



Gq = mql3q = \^.Jm^l3q + ' 

to give, for t > 0: 

^X\y{\\Cq- tq-¥.[Cq- Cq]\\ >i} <2QcXp. 

which, using the triangle inequality 

||lA|l-|li?|l|<P-B||, 

gives 

fx\y[\\Cq-tq\\ > t + \\Ex\y[Cq - Cq]\\^ < 2Q CX^p ■ 



QM 



We may want to bound HExiyi'Cg — C,q\\\- To do so, we note that for any i such that tji = q, and for 
distributed according to Dx\q' 

^X\y^q = ^X\yLq{Az, X, y) 

^x\yL{Az,X,,q) 
= ^ ^xxi\yL{Az^,Xl,q) 

= ^X.Xi\yL{Az^,X'^,q). 

Hence, using the /3q stability 

||E[£, - 4] II = \\^x,xi\y [L{Az,Xl,q) - L{Az^,Xl,q)]\\ , 

< ^x,xi\y \\L{Az,Xiq) - L{Az^,Xl,q)\\ 

< Ex,x^\y \\LiAz, Xi q) - L{Azx.,Xl, q)\\ + Ex,x^\y WHAz^ , X^, q) - L{Azv , X^, q)\\ 
<2/3,. 

This leads to 



^x\y{\\Cq-Cq\\ > < + 2/3,} < 2gexp. 



1 .m 
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Step 4. Union Bound Now, using the union bound, we have 

F {3q : \\Cg - Cg\\ >t + 2/3g} < ^ P : ||/:, - 4|| > f + 2/3,} 



< 



2QE 



exp ■ 



q [ 8(4Vrn^/3, + ^)' 
< 2Q^ max exp 



+ J 

According to our definition of confusion stability (cf. Definition [5]) , Pq decreases as Therefore 
P |3g : \\£q - Cg\\ > t + 2/?q| < 2Q^ maxexp 



where m^i^ = minqiriq and /3min is the associated /3q. Setting the right hand side to S, we can get, 
with probability 1 — 6, 



^ ||C« - Q,„p|| <2j2Pq + qJ» 111 f ^ ) ( 4>^/3„,i„ + M 



q g » ^ 

Hence the result. □ 

4 Analysis of existing algorithms 

Now that the main result on stability bound has been established, we will investigate how some already 
existing multi-class algorithms display some stability properties and thus fall in the scope of our analysis. 
More precisely, we will analyse two well-known models for multiclass support vector machines and we will 
show that they to promote small confusion error. But first, we will study the more general stability of 
multi-class algorithms using rcgularization in Reproducing Kernel Hilbert Spaces (RKHS) . 

4.1 Hilbert Space Regularized Algorithms 

Many well-known and widely-used algorit hms feature a minimization of a regula rized objective functior lTikhonov and Arser 
(1977). In the context of kernel machines Cristianini and Shawe-Tavlor ( 20001 ) . this regularizer fl(h) may 



take the following form: 



where k : X x X ^ M. denotes the kernel associated to the RKHS 7i. 

In order to study the stability properties of algorithms, minimizing a data-fitting term, penalized by 
such regularize rs, in our multi-class set t ing, w e need to introduce a minor definition that is an addition to 
definition 19 of iBousauet and Elisseefi (2002). 



Definition 3. A loss function I defined on TiP xy is a -multi- admissible if £ is a -admissible with respect 
to any of his Q first arguments. 

This allows us to come up with the following theorem. 

Theorem 3. Let % be a reproducing kernel Hilbert space (with kernel k) such that VAT G X, k[X,X) < 
K? < -\-oo. Let L be a loss matrix, such that\fq e y, £q is Uq-multi-admissible. And let A be an algorithm 
such that 

^5 = argminY] Y] —^^(/i, x„, g) + A Y] := argmin J(/i). 
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Then A is confusion stable with respect to the loss matrix L. Moreover, € y, we have 



2Xmq 



Sketch of proof. Roughly, the idea is to exploit definition[2]in order to apply the theorem 22 of Bousauet and Elisseefj 
(12002) for each loss £q. Moreover our regularizer is a sum (over q) of RKHS norms, hence the additional 
Q in the the bound on /3g. □ 

From now on, we will always suppose that we are working with kernels such that k{X, X) < < +00. 
4.2 Lee, Lin and Wahba model 

One of the most well-kno wn and well-studied model for multi-class classification, in the context of SVM, 
was proposed bv lLee et al.. (,2004) . In this work, the authors suggest the use of the following loss function. 



Their algorithm, denoted A^^^ , then consists in minimizing the following (penalized) functional, 

Q 



with the constraint hq ~ 0. 
We can trivially rewrite J{h) as 



1 ^ 

■^W = E E —e,{Kxr.,q) + xY,\\hX^ 

9=1 



with 

iq{h, Xn, <?) = 51 (f^p(^k) + 77—7 

It is straightforward that for any q, Iq is 1-multi-admissible. We thus can apply theorem [3] and get 

^9 - 2Am, • 

Lemma 2. Let h* denote the solution found by A^^^. Vx e X^y E y^q, we have £q{h*,x,y) < ^ + 1. 
Proof. As h* is a minimizer of J, we have 

J{h*)<J{0)^J2 E — ^9(0,a;„,<7) 

= V V - = 1. 

^ ^ (Q- 1)771, 

As the data fitting term is non-negative, we also have 

Jih*)>xY.\\K\\l 

q 

Given that h* G "H, Cauchy-Schwarz inequality gives 



\h*(x)\ 



Collecting things, we have 



\fxex,\h*qix)\ < ^. 



Going back to the definition of Iq, we get the result. □ 
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Using theorem [21 it follows that, with probability 1 — 5 

Q 



Xniq \/'71,nii: 



With regards to the f3q we obtained, one can conclude that ^llw confusion stable. 
4.3 Weston and Watkins model 

One of the oldest models, when it comes to multi-class SVM is due to IWeston and Watkins ()l998l) . They 
consider the following loss functions. 

£{h, y) = ^ (1 - hy{x) + hg{x))_^ 
The algorithm yl^w ixiinimizcs the following functional 

m Q 

This time, for 1 < p, q < Q, we will introduce the functions hpq = hp — hq. We can then rewrite J{h) 

as 

-, Q p-i 

•^(^) = E E —^qih,xn,q) + \Y.J2\\''pi\\'^ 

q n:y„=q p=l 

with 

iq{h, Xn,q) = E ~ ^pqi^n))^ ■ 
p¥=q 

It still is straightforward that for any q, (,q is 1-multi-admissible. However, this time, our regularizer 

4Ar7 



consists in the sum of '^^'^ < ^ norms. Applying theorem [3] therefore gives us fiq < 



Lemma 3. Lei ft,* denote the solution found by . Vx G X,yy G 3^,V(7, ■u;e /laue £q{h* , x,y) < 

This lemma can be proven following exactly the same techniques and reasoning as Lemma [5] 
Using theorem [21 it follows that, with probability 1 — S, 



With regards to the /3g we obtained, one can conclude that confusion stable. 

5 Discussion and Conclusion 

In this paper, we have proposed a new framework, namely the algorithmic confusion stability, together 
with new bounds to characterize the generalization properties of multiclass learning algorithms. The crux 
of our study is to envision the confusion matrix as a performance measure, which differs from commonly 
encountered approaches that investigate generalization properties of scalar-valued performances (such as, 
e.g., the accuracy). 

A few questions that are raised by the present work are the following. Is it possible to derive confusion 
stable algorithms that precisely aim at controlling the norm of their confusion matrix? Are there other 
algorithms than those analyzed here that may be studied in our new framework? In particular, is it the 
case for k-nearcst-neighbors, the generalization analysis of which is amenable thanks to classical stability 
arguments? On a broader perspective: how can noncommutative concentration inequalities be of some 
use in machine learning? 
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